Skip to content

ci(scient): quarantine exactly matched flaky Oh My Pi test failures - #76

Merged
yaacovcorcos merged 3 commits into
mainfrom
ci/flake-quarantine-20261118
Nov 18, 2026
Merged

yaacovcorcos merged 3 commits into
mainfrom
ci/flake-quarantine-20261118

Conversation

@yaacovcorcos

@yaacovcorcos yaacovcorcos commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

What

Scient Desktop's flake-quarantine rule (owner decision, 2026-11-17), applied to Oh My Pi's test buckets.

  • scripts/scient/ci-quarantine.json holds the rows. Each names one test, a signature (text the failure's own output must contain), an owner and an expiry 7 days after quarantine.
  • upstream-tests.ts's judge lets a failure through only when an unexpired row names its test and the signature appears in that test's own output: the lines between the previous test and its (fail) line. The same test failing another way still fails the job. So does everything KNOWN_FAILURES never excuses: a crash, an error outside any test, or counts that don't add up.
  • A quarantined failure is printed as ::warning:: and listed in the job summary with its row.
  • A release run (RELEASE=true) takes no rows, so a release ships only from a run without a quarantined failure.
  • KNOWN_FAILURES (exact-upstream failures) is unchanged.
  • SCIENT.md › Checks › "Flake quarantine" documents the rule and the register table. A test checks that every row is unique, at most a week long, and in that table.

First row: browser-open-options-downloads. "saves each tab's downloads into its own downloads directory, also after another tab closes" fails when readdir(second) finds a stray <guid>.crdownload.

The other two 11-16 browser tests named for this register need no row:

Why

Since #69, main requires CI passed, so a flaky upstream test now blocks an unrelated PR until someone reruns it. This test failed on main 11-16 and on docs-only PR #70, both with the same stray .crdownload.

The cause is open. It is upstream's test and code (downloads.ts), unchanged in Scient. Evidence so far:

  • In 30 local runs against Chrome, Chrome's own download never left a .crdownload at completion.
  • Chrome did save downloads that no page started (no Browser.downloadWillBegin) into the folder it was pointed at last, the second tab's in this test.
  • The flake has not reproduced locally (10/10 passes), so no fix is proposed.

Testing

  • bun test scripts/scient/upstream-tests.test.ts: 13 pass. New cases:
    • a matching failure passes and is reported;
    • the same test failing another way blocks;
    • a signature elsewhere in the chunk does not count;
    • an expired row, or no rows, blocks;
    • a row with non-canonical dates or a span over a week never matches;
    • every register row is valid.
  • The judge replayed on both real CI logs (jobs 134034042944 and 133617394015): today it quarantines the failure with no other problem; on 2026-11-26 the same failure blocks.
  • bun check: passes.

Platforms

  • Linux: the only place Oh My Pi's buckets run (upstream-tests, ubuntu-22.04). The change applies there.
  • macOS and Windows: unaffected. Scient's own macOS tests and the build matrix don't use this judge.

Review

Review: Light Codex (gpt-6.1-sol, high) — 1 finding (malformed expiry dates compared as strings could keep an expired row in force), fixed in 023bddb with a regression test; recheck clean.


  • bun check passes
  • Tested locally
  • CHANGELOG updated: not user-facing (CI only)

🤖 Generated with Claude Code

yaacovcorcos and others added 3 commits November 18, 2026 07:33
A row of scripts/scient/ci-quarantine.json names one test and a signature
from the failure's own output; a matching failure passes the bucket with a
warning and a job-summary line until the row expires, 7 days after it was
quarantined. A release run takes no rows. KNOWN_FAILURES is unchanged.

First row: browser-open-options downloads (a stray .crdownload in the
second tab's folder), seen on main 11-16 and PR #70 on 11-17.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A row's dates must be canonical UTC YYYY-MM-DD, expire no earlier than the
quarantine and within 7 days of it; otherwise the row never matches. String
comparison of "2026-11-2" kept an expired row in force (Codex review).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yaacovcorcos
yaacovcorcos merged commit 7b91cb1 into main Nov 18, 2026
22 checks passed
@yaacovcorcos
yaacovcorcos deleted the ci/flake-quarantine-20261118 branch November 18, 2026 06:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant